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Title of Invention: A method of hierarchical digital video summarization 
and browsing 
Applicant: SHARP INC. 

Disclosed is a method of hierarchical digital video summarization and 
browsing. The method comprises the step of: inputting a digital video signal for a 
digital video sequence; and generating a hierarchical keyframe summary. The method 
further comprises the steps of: computing histograms for the digital video sequence; 
detecting shot boundaries within the digital video sequence; determining the number 
of keyframes allocated within each shot; locating the actual position of each keyframe 
within each shot; admitting keyframes by the largest consecutive difference criteria; 
pruning keyframes for a shot without meaningful action; extracting keyframes with a 
better efficiency in case of a compression video; and browsing a shot using the 
hierarchical keyframe summary. 
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R13 6H Pg^^^co^-^U-A^^'J^, 0H*.tf* BfMM^S^HiJfT 

t^h^^^M^nix!:, fe b @W^-^l/-A0ii^\ H4£#HBb«^ 

[0 0 6 7] 

*L<Dm$fmmttim 11 »Bv-^r>^*0^ f ^*/;pyffl«0^ta (Detect 
ing Dissolve Regiors in Video Sequen 

ces) " izm^Lrcmm^^ormm^^^ v^m^mihmm^. txh 

^^A^Itlth^^hXh'J-AOT^y^Xr-^ (BIT) (0^^ff3 
07 (Df&m^Uy 2 1 3 4\Z&miZ&fr&tS££.i)t~C£Z> 0 IP "6, 5=V V)V~7M 
7U-A£B I TfttCV— ^b, 7 1/— ABJR^D-fevU:: 

^^Tiffit^o — A*<8© + — — A{C 

bTfcJ;lio 

[0 0 6 8] 

tXh £^;7 AGOffl? 1 3 4 e7hXhU-Art^Hh 
*:7-tzyHaB, #fis^l/-A, D C p 7 l/fcifft^ b 'J 

"TS. **WtC*3ViT, "tf^ HJ-A-f >xy^^f-^" (BIT) 
tfo-rZ^-yjl&tkf&'TZe B I T 1 3 8©3>f>7iB I TO^^ffitC 
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B I T 1 3 8<Dl-0<DBm** tfy HX h U - A ©*M Wft/^ 7 ^ - i' Sftfi 

^T£5tC, g^Jc^^Ui^SliBt^y hX HJ-AfCjjn^B I TX^B I T<DM 
/Jvt~^ 3 >£g«rr*. 3.-iffcB#5l*«^b, ST(D»iS**«>fc**ffi© 

ui^Tt^o B I T£2 6t-n>/\°^ HcTS (XtiffJ«-TS) ^ffi^OViT 
yOK7)£j3r <Z)*T?Ift9rr§o B I T^MIB I TlMXtf) "B'JM" 

[0 0 6 9] 

2^b i T\z^^nx^^mm^nmv^r—yv-^mumz^ni,xupR 

G-2fl[^»l 4 4^310, «^^C», 4 0£ffiWE-r*. **JgV* 

HK-TSilfc^atflB^, P3«fc*X^£#5fr-rs. «^»^*-^*©f£IMH 

[0 0 7 0] 
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#5B9itt, mpeg- 2!£mmw&wm^Tnm?zmmt}A7\H\zmm~cz 

[0 0 7 1 ] 

9J3g£bT, ¥y hX h >J -A£# ^ <=>S^:/ci-feX£itfT T 3 3 > bTzL 

X hU-A^/\°-y^-^3>t:zL— ^ (PC) t:^>p-R^§ 0 
->X^Ate, PC^77 hX HJ — A £4fl31T 

[0 0 7 2] 

MPEG-2t'7hXMJ - A £2&spcfc < J!iab|SiFSM»ilii5#9«rf^fi!c"r Sfc 
J^Tc^F^^^^Dffl^^b^^n^e^v^ 

1. MPEG-2 tfy hXhU-A^IMI^l^f), 7U—ACOBH 

2. 4t©7 U-ASSf 1 5 Jli:^<MPEG - 2 tf-/ bX h'J-A^^M 

b&*-:7 u-A^a^-rs^jesaii-rso 

3 . 2 «c»Bf^»^r— 7 U-APb1<Z)— 1©7 U-A&fi^TSWlBSSrBB^-rSo 
^OTjffite, hXh^7AK;i/Tl< 0 MPEG- 2 b:y hX h'J— A£« 

/hisa#b^e>7u-A^jcfetx h^7A&»»T5*ffiSH*rs e 

[0 0 7 3] 
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l»l:ttt»T»o^ »£*i«iliffc©«£\ st&m&WMTZfctblz* tt^co 

Tl/^o MP EG^^&tC^V^nTl^^D >^+- AtCfi£(/^ 

16x1 6T^O^D^Ol^ ^fT*0»€^«^«l<B#!!H:7U 

— AC9 1 6x1 6 vi7D^D^^o«^:E4r)<7>v^D^Dy^ < hS^a^fc : B:5 
o mWz. 16x1 67^7D^Dy^f x 8©H-^P7^*«*I840 

©ffi© 8 x 8 it^ ^ D 7 ^ tlfe^totSo fr&olZ* &8x8+)-77uyp$: 
, fifr£#8 x 8lf:/:/a*;/^*<Bffi<0fi*tt^ 

ct^-ctfc. H9£#flsu 8x8^?D7^©¥^fin *3;~eftff2n 

-So 

[0 0 7 4] 
[St 5] 

((flXAXw, )+(8-a)(A)(w 3 ) + (fl)(8-6)(m 2 ) + (8-a)(8-6)(w 4 )+ )/64 (5) 

[0 0 7 5 ] 

Adct yp 7 ^ c^t^liD c T(?)D c^<z) 8 ^> (m 
p e g - 2 mmzm i^ns ^ & \ztt~rz>m^D ctodcsm^d ^ ^ 

©SSKDa^ffi© 1/8TS5OT, 8CDfiU8to*A§) o t^h^AH 
^S^lZ&Ztl&^OlZ, Hftf^lC0#8 x 8 7 , P7^WttX K^A^ 

iT^fefe^-fW^tlTt^o *»^o^ffioacfi©lo«, MPEGH7 
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[0 0 7 6] 

MPEGb'7 m-A^g>(Ot:7hy7A<OlH 

MPEGti7 h 'J— ACOlS^te, 2 O(0lt JE^ittS^r^ ^ 

1. 8x8yD7^^*DCTm o 

2. MPEG-2(Di^ 1 6 x 1 6 Dt/O y £ tC<fc§»£«*. 7Uy? 
5telc, 8 x 8yn^^* J E-cC)¥^fliTK$SlAT ! bi®«cot7 k b^Atc*# 

sx (DCTMoDcti) tbist*. »DCTn-g:cD^is6co^s:^6, 

[0 0 7 7 ] 

*©xf7^1»t§fcfei:, MPEGif^hxh u— AtcSjasnr^s 

^fb«BS^^^T-€-<D«lll&S:SlWrS. «»MP EG If y h 'J -AH 

I (^U-Artft^fl^lx-A) , 
B (5K^^»J7U-A) Rtf 
P (^SI^U-A) o 

fit* h^7AC0ft3KcDCT«SR<Z)DCfi«:ffll^i: I :7U-A<B«4MS« 

^(Dmmiz^izMzk^^ z>» b ^ u-a chp^ u-juitVcizmm- fz»m y u 

-f^tS. ^^^^L^^Uffl^^^U-AS#HglS^WiC<0^^$nT^^ 

T*a^ffli^nTi^»$}ffli^^-AicHb, ^t-t; (b) \tcz.-cm^-r 

S«rU^ft€rffl*X*-AtcllB-rs. 
[0 0 7 8] 
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— BP'S* §8x8^07^7^, Wl^/;8x8WD7 
2<Dmfr&tttftz¥-t%m-eW%WkZ-Z>o 2 0©ytU*SMn. ^-X (a 
) te8 x 81f^U^^^ffl»W(ca^bfc#fi97U— A + OfiJiofc^P^^ 
(Bfi^*tt#fc¥^fiTB€f«A*. ^r-X (b) \t. 8x8WD7^«, 
^W^^^bfc#^^U-A^6cOiE^^lij^TS^^A^o ^r-X (a Ml 
43^T, K^ffifS^S'J^l/-A^c7)^:8 x 8 rfu y ^^HI^^o^ £^¥U£ 
. ^r-X (b) K^Tte, 8 x 8 7Uy?\m&mz&<<D^t£Z>m$:mi5m 

T^/c^tC, ^r-X (a) <h^r-X (b) SILT 8 x 8 y 

£<, aiOIJ^OWSSLTV^. H10tC*3liT, T-iJjyo^^^ I 

SP^, 8 x 8 7nv?te^M7U--I±tp<DM&TZ>&— ffi£8rofc<Z> 
<hfe^T£o ^r-X (a) fe« 1 ^©i/i 1 ^© 8 x 

8^7^*4^. tr—Ts (b) \tm^Emz4-D(Dmu^>m^:^^myu- 

AF*K0lO©8 x 8^D7^^i;^o 
[0 0 7 9] 

(P^diB7U-A) tc^&M^U&^o h 

L^L^^e, y-X (a) {ifhff»^^<^^U©?H»fe^^^TJ;D» 
SU^o Cttte, ^r-X (a) ©J;3{C»^M*$tlfettac9f^7^-A (I 

xupxtSB)' ksshtw:, &8 x 8^Du/^Jctifc/f io©ffi^f#e»n^>^e> 

^7U-A*»«rs»^tlt«UT»ffia9^1ittiUTf4, 64 (8x8) » 

(C, y-X (a) ^ifiS^ttffifiTSr.fefc^-rWT?, ffiffl^ffi&ft*T?teft^o 
[0 0 8 0] 



(35) 



2001-527304 



yu-AmmDwr^t. y-x (a) <h^-x (b) moimmtt&o m3 

CMot, lfo<Dm$ffiMm&7U--l±frt>^W}$:, A^6B 

s^4#»t§o ^-x (a) <b«£\ m^frt^ftaias^a'j^ny^^s 

8 x 8<^^J^n^^^lr^bm--c0{gT : ^7"r^ o ^-X (b) 

A^e»©ZKO— a©^aO<i:LTy— X (a) <h^r-X (b) M{CffiST§fi5^: 

X/OH^KI^I^ ^-X (a) te, «ait"S«t'5^7U— AMcoK^^+»t? 

^r-X (b) trte^U&l^ 
[0 0 8 1 ] 

™x (a) iz^xmmi^rc^oizm^mmit, c0/MMiB©2x2¥i5 

[0 0 8 2] 

±m<Dmm$m^&%ct$mmzwimznrz 0 jiisw&mpeg - 2fflts 

->--y>^cD»^.2tt©l7 1/--Ara<OEll|[j:l 5t?*5o 

ffi^^y-X (a) ^av^fr-rsi:#*^K*tttt««T^*i;Si:i:^»^ 

Stlfe. ^-X (a) ©»#««X+-Attt^h4f7A^H»W^!fi^JK«iS: 

X (b) £^3glCftJBL;fco IM^S^fflt^Z h>/^ASr»»f Sdt^J: 
0, l^QC I F->— ^r>XK*f-r§fcX h^^Att^flS^^^filfilSn 
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h^A£ 12 8 ^/©U^tXh^^At 1 2 8 t f ><DV«»t7 h^7A 
[0 0 8 3] 

MPEG— 2 \iy H X h U — Afo SflUftBU? V— AOPffifl 

^dtrP^^T^^SS^tC^^T, MPEG- 2 fcfy hXbU-A^^^ 
^U-Ac^amtl2X^^y^^|llTfT^n^ 0 tX K^^AcDft#chfW]0#tC|| 
fT^tl^mi X^yZflZte^T, MPEG- 2 tfy h U - A^ <o 7 >^AKI 

>5^^X^->0]/' ££j$f£o +-7l/-A(OBi^4j5S$nS. IP'S, 
U^;U©:7U— AOI^-fey h«h&5o ^— 71/- AffltB#»Htc43ttSS2 A-r- 
[0 0 8 4] 

Jif2 2 7,7 : -^^)[icofiJ^J^ b'^y hX hU— A©B8affl»*fi«fi#"rsa 

*ffl«T§5CtT*-5o 71/-At^l 3 5 0^6 1 4 0 0 ^TOMPEG- 
2 t + y hX h U-A^t:^ hX h U - A-f >T7 ^Xr-^^ffl^f ^I^t 
^/i^te, 7U-A#^1 3 5 0 HX h U-A^, ^±lzm^fL 

o t*7 hX MJ-A-f >f 7^Xr-^M^ITtnH IfyhXh'J-A© 
[0 0 8 5] 

MPEG- 2 tfy hX hU-A^b^>^AfcMtRb^7l/-A (1:^51 
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1*^2 0 0 1 - 5 2 7 3 0 4 



y h) o 

2. S7 I/ - AO t # 7 b ^ h U - h * 7 1 7 h . 

3. aiKfc^T (^co;t-r h^^iry H Bfci6^» J Hfc;v h 'J ^X^'J-t: 

4. A^B^P— AT?»ntf, 5E:7U — A{£#JST£#BB:7U— A (I 
-P/I-I/P-P) »ne><07l/-A®;Hht7t7b) - 

5. a7l/-A^P7l/-At*nH (P ^7 1/-A©#17 1/ — ATf&£>) 
t>i£^ I 7 I/— A (■£-00:7 1'— h*7±y h) 0 

[0 0 8 6] 

B/P7u-A0i^t 7i/ — a ^ ieas iz m^^> tz # , m&yv- 

tftfB2OC0X^^y, (1) t^yhXhU-A-Ofy^Xf-^^ 

^fiETS^^^yi (2) ^ NX h'J-A^> 

OTfBOii^ > h L fcfy MJ -A-f ^X^-^K7)^tcffl 

[0 0 8 7] 
[HI] 

cnumlndexFileState { 

K^SEQUENCE„HEADER=0, 

^PICTURE JFRAME, 

KJPICTURE.BFRAME, 

K_PICTURE_PFRAME, 

K^QUANT_MATRDC_EXTENSION f 

KJEND_OF_DATA, 

K_OEFSET 

}; 
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[0 0 8 8] 

K_0 FFSET77 tf\Z, ±E©ffi© 77^S fc* leffiSA-f h 
*7-fe^y HJCJUAS, 1 5GDA-< bit. ft* OSHSTte 1 5 + K_0 

FFSET (=2 1) Lmm~?Z>o K_END_OF_DATAH KftS^ 

[0 0 8 9] 

1. y-^r>Z^7^0/HhT« 

2. I tr^^aU 5 OA-f h 

3. Pt^ft^3 0 0 0/Hh 

4. BWft^4 2 0 0/Hh 

5. BWft^5 3 0 0/Hh 

6. If ftv h U 7^X(Z)Ut7 4 0 0;H h 

7. Pfc°^^6 2 0 OA-f h 

[0 0 9 0] j\ 

K_END_OF_DATA K_SEQUENCE_HEADER K_OFFSET+0 K_END_OF_DATA 
K^PICTUREJFRAME KJDFFSET+150 K_END_OF_DATA 
ICPICTURE.PFRAME K.OFFSET+3000 KJEND_OF_DATA 
KJ?ICTURE_BFRAME 

K^OFFSET+4200 K JEND_OF_D ATA KJPICTURE_B FRAME K_OFFSET+5300 
K_QUANT_MATRIX_.EXTENSION K.OFFSET+5400 K_END__OF_DATA 
KJPICTURE_PFRAME K_OFFSET+6200 K_END_OF_DATA 

[0 0 9 1 ] 

[0 0 9 2 ] 
[fS3] 



(39) 



#^2 001-527304 



5 0 6 5 1 156 5 2 3006 5 3 4206 5 3 5306 4 5406 5 2 6206 5 

[0 0 9 3] 

tfrtSfe»^ST»5. K_END_OF_DATA77^ll *S2Bim*tf 
Tl^T&^o £C7):7^te, *£Lfc«(B7 i -*£l**T£fc# 

— (c»L3S<-r^- CO^^^tift^— ^^^*-rs/ti6<7) "HSHI-Sf" ill 

[0 0 9 4] 

«TflB© J: 5 tc«ffif 

1. mClk T ^^n^7l/"A©#^ (ldf) ItcSJSHft-rSo 

2. ymznz&yio-j* ( f t d) : 

ffl-^S:BBS6"rs^U-A (sdf) ^lottH11"c 

f t d^I 7 AT?*ntf * s d f = f t d <hT£o 
f t d^P7l/-AT*nH s d f £ f t diOmKOSfciff^ I 7U- 
At£-fey bt*£o 

f t d^B7WAT^nH sdf^f t d izM^^T >ft — 7 V— 
±IB sdfj&Mdf + UO /J»S tfntf , s d f - l d f + 1 tCl^^T^ 

• sdf<idf + it»ntf, m&2titz7u-i±<Dm-z>fri*'T'e\zm^n 

i = ldf + l-sdf tC*fLT, 
«fcjev^>— y>XA, y ^ (r s h) &M*~3ttth-1r* 
qmr^r s h £ 9 ^*§^te, IfeifitiifftvhU^^XiOiJty 
h (qmr) ^Ott£tlt (qmr^Sffitntf) . 

rshiqmr^h'^hXh U -A^tcaSTaJB/fT^T*,, 
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f t dSS^tS. 

171/-A (f td) Sra^T^o S^71/-AH, ia(D7U-A<0««* 
[0 0 9 5] 

AfC 1 7^— ;i/hM£^LT^£<DX:\ tXh^7ASttRl/!5:<X<fcK — 
^H^^U-A^eiSSULja^J^X^^y^i:^^^^, MP EG- 2 bf^ bX 

2. (br^^^«F*©S63B*©) h*^^^\ h77 r /*hA7>r-^Hi 
[0 0 9 6] 

■J— A-f >x^^X^"^^3>/N°^7 blcSl^-f ££<hT&3„ MP EGtffl 
7 I/ - ASft 0 fi 8 - 1 0 S i: 5 lt&©T, Ztlfo, — 
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1. ht7t7 hJ;f9t)li^m h^7t7 h^l/^o ZtllZi:?)* 

2. Un i x^77 b^— AcDg z i ptfeBPC^pk z i p <£> J; 5 

3. K_END_OF_DATA77^Pli« 0 

[0 0 9 7 ] 

3--if^T^MT^^«t*IT$^:V^t{Cffi*T^STftS^ (1) t (4) 
, (4) cZ)M'JST;i/=/'JXAH, 7 ACD^HJlI/^£Ml&T£cfc5tcm h^- 

[0 0 9 8] 

if 7 h u - a< >f 7 ^xf-^ (bit) tmvmm&m^&tf* m 
mizmm^y h7,b'j y^y^^-y"^ (pbit) ^(o^^iiy 

nTt/> HIV AtC^ffaU (K_END - 0F_DATAO77^7?R 
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[0 0 9 9] 

1. S*— ACO^-f :/I/B/P 

2. 3K+-7U-Alc*fJE-rsBBJ6a^7U-A (sdf) 

T©+-7U-A&«^TSfc*tCttie^»b«:^©T?* (sdf<ldf + 
lTftntil d f + lT^t^) AW h*7iry h^<D^m<Ds d f te^WT^ 
^S^*So tg^^li^IiU s d f < 1 d f + 1 T*ft*l«, 1 d f 

+ ia>SIIB&T££<ha*#BT*5 0 1 d f + ltfftff*—7)s—2x\Zlgt<thm 

4. J£fcifiV^>— y>A^ y ^©^7iry He ^T©*MMPEG-2t*7h 
X h 'J-Ate>— ^r>7^7^tl/Tls50T', K_SEQUENCE_HE 
ADER77 ^tei&Bft^., 

5. *HbvV U ^^XCD'Jiz^ MPEG - 2 ti7 hXhU 

— AtCtia^fbV MJ ^XUizy hT^&^CDT, K QUANT MAT R I 

X_EXTENS I 0N77^MttTim-M^^^ 0 i/— f T>7s^j9 

[0 10 0] 

afs-7*-^t^ s d f ««<ts+— 7ix-A©^-r^i:-5-<7)A>f 

a^Sv^-v^^Jis d f ^S«^£§Bi6U t> hXh'J-A^ISWU 
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^S^-i/>fis d f ^e>J6*S#^U— ASr*aLb, ±X<D I &L*P 71/- 
[0101] 

^71/-A^B^^^TfenH B7I/-A*iX»fft5^I**<ffc 
^(C, «fci£l>«^#jI8 (IMP) 7 1/- A\Zj&M^i±Tz^ 0 B7 1/-A*5 

3 5 0 OA-f h<D*y-ty MI J; Ote^^^^t*^^^— 7 I/- AT* 0 , 4 3 
0 0 OA-f K^b^tTS^fbV h 'J 3 9 0 0 OA-f K&@T' 

4 5 0 0 0/H ^CO-^-^^^cOcfc^) ^^-^^^nTl^o 

[0 10 2] 
K4] 

KL.END_OF.DATA KJPICTUREJBFRAiVlE K_OFFSET-h45000 
K^OFFSET+53500 K_OFFSET+39000 

K_QU ANT_MATRIX_EXTENS ION K.OFFSET+43000 KJEND_OF_DATA 

[0 10 3] 

a^g^-^toajifttPB i T©*>f X(t©ra 

h l/- H^7^*a 0 ffl^©B**fl=JC J: OjSSj&A^ >^ & <h 5 d £ Bjffi 
T*So ffil^-tf, PBITiisd f safta+^Iz-APflOitfOIS 

PB I TO-y-rXttii^cTS. ftS<BlSS8ft«£fct, s d f^-7 1/-A 
©A-f h^iry h<B*S*Brr£. St>3>Ai7 hSPB I T?Mf sn£j&«, 

lft/N°>/X-A^mXfflm«^^^3Atr^^tC^, ±7U-At*y hXh'J- 
ASa#t5^*S. 
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[0 10 4] 

o MJ— Ate. iiffir©-?^ ^^;HJiHT*S^, ^te, DCTt:S 

[01] 

0 1 te, 3:2©U^;WrMTS»nIic^^ 
[0 2] 

m 2 ti, *«wicd:s^ffi<o» i mmmmcD-fn y $ m-?s&z> 0 

[04] 

[0 5] 
[0 6] 

06H *-7U-A (kj) *3«ttf»J6>'3 7ht^>h (tj-i, tj) 

(Dmmm^mmm (c ( X > ) 

[0 7] 

m7\z, mpeg- 2 \z&*j£&7<tiWim\zmm2nz*ftw<om2nmj&& 

[08] 

0 8te, ^WS»^<Z)^— ^l/-AO«^tCffil^nSx-^©^T0T^ 

So 

[09] 

0 9te, Bfraioy77T*-5. 
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[0 10] 

mi ota, >x-7s (a) Rtftr—Ts (b) ^m-r^rz^izm^^m^mmy 



[01 ] 




o 

CO 
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[04] 



80 



,38 



,42 



82 



{t0,tl,t2 ,tK} 



.84 



> 


{k1,k2,....kK} 






r 


SB* U ^KZ> df- — 7 u- — A 



,86 



.44 
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[07] 




O 
rO 



[H8] 
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[010] 





i 120 


4 i 


126 






! < 




















124 




128 





A Dcr°™ 



(a) 



(b) 




M=(4X4X12<H-4X4X126+4X4X124+4X4X128+124y64=126 



ml 


m2 


m3 


m4 



ml = 120 + 124/64 = 122 
m2= 126 + 124/64 - 128 
m3 = 124+ 124/64= 126 
m4 = 128 + 124/64 = 130 
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[lHi0]¥«ll¥ll^lOB (1 9 9 9. 1 1. 1 0) 

i^mmxE i ] 

fulfil s^;u»Hv-y>ztc»T5fcx h^A£ft»T3A^;/:/£, 

[*tIE#*««*] eJifcfflS 
[*tIEl*i*] 



v (55) 2001-527304 



[HUE**] ^JE 
[#M«IE4] 

[|f^2 0] HUBex-C v'^^iftHv— ^r>Xtt. m&S^tlfcT^ v^l- 
« 1 CIB*©»0W'7^ S?^;HfcHBtt&^BME*fe 
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